[Feature] Let the complexity signal consume a remote backend (score.v1 + label_distribution.v1) - #3542
[Feature] Let the complexity signal consume a remote backend (score.v1 + label_distribution.v1)#3542theohsiung wants to merge 11 commits into
Conversation
✅ Deploy Preview for vllm-semantic-router ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
The shared backend block reached only the category signal: `CategoryModel` was the one struct with a `Backend` field, and the contract whitelist held one value. Complexity could not name a remote model at all, which is the gap vllm-project#2921 exists to close. Complexity accepts two response shapes, because a difficulty model can be built either way, and the contract chosen decides how the runtime reads the answer. `score.v1` carries a continuous score; the existing `label_distribution.v1` carries the verdict as a label. Neither substitutes for the other, so this widens the resolver to take every contract a consumer declares. Passing exactly one keeps it available as the default for an omitted field - category is unchanged - while a consumer accepting several cannot default: guessing wrong would surface per request rather than at config load. The backend attaches beside `prototype_scoring` rather than on a rule. `routing.signals` is replaced wholesale per recipe, so a backend declared there would vanish under any recipe that did not repeat it, leaving a signal that still runs but has quietly reverted to local. A score in the model's own units cannot be compared against `threshold`, which is symmetric because the local margin is signed and centred on zero. A rule can now state its two boundaries explicitly, and the pair it uses says which way difficulty runs: hard_above with easy_below where a higher score is harder, hard_below with easy_above where a lower one is. Encoding direction in the names keeps a separate direction field out of the schema and makes an overlapping band impossible to write. `threshold` stays valid as the symmetric shorthand. The reference config gains a rule using the explicit pair, since the asymmetric form is useful to the local path too. The lower-is-harder pair is excepted from reference coverage instead: the local margin is hard-minus-easy, so a higher value is harder by construction, and a local example would document the wrong direction. Signed-off-by: theohsiung <theobear870924@gmail.com>
ScoringBackend has been declared since the three-tier backend split but carried a note saying nothing implemented or wired it. score.v1 is its first consumer. It cannot reuse HTTPClassifierInference. That type exists to align a label distribution onto a mapping, so it requires one with at least two labels; score.v1 has no labels at all, the score being the whole product. This is a separate, thinner reader over the same connector, which keeps timeout, retry, byte caps and error mapping shared rather than duplicated. The wire shape is the HuggingFace text-classification array, because score.v1 shares the http_classify protocol and endpoint - a regression head deployed behind the usual /classify needs no shim. The label is meaningless here. Anything other than exactly one entry leaves "which is the score" undefined, so it fails rather than picking one, and the value is returned unclamped because a regression head is not a probability. Score takes the caller's context. The interface had none, which is the gap the category backend had to close after the fact; starting with one avoids repeating it. Nothing implemented the interface, so widening it breaks no caller. Signed-off-by: theohsiung <theobear870924@gmail.com>
The direction pair was structurally validated - no overlap, no mixed directions, no half-declared pair - but nothing tied it to where the score comes from. A local rule could declare hard_below/easy_above and be accepted, then invert every verdict it reached. The local margin is hardScore minus easyScore, so a higher value is harder by construction, not by convention. A local rule declaring the opposite direction would score the user's own hard examples as easy, and the config would look entirely reasonable while doing it. The inverse is already expressible locally by swapping the candidate lists, so the pair is rejected there rather than honoured. Rules inside a recipe are checked too. routing.signals is replaced wholesale per recipe, so a rule that exists only in one would otherwise skip validation entirely - the same reason the backend itself is declared on the module. Found by asking what actually stops the semantically wrong combination, after excepting those two fields from reference coverage for exactly this reason and not carrying the conclusion into validation. Signed-off-by: theohsiung <theobear870924@gmail.com>
Both remote contracts now produce the complexity signal, and the local prototype path is unchanged when no backend is configured. The two paths return the same rule-result shape as the local classifier, so the publish loop stays shared: match names, metrics and published values cannot drift between local and remote because there is only one place that writes them. score.v1 makes one call per request, not per rule. The score is a property of the request; a rule differs only in where it draws its boundaries, which is why the backend is declared on the module and why five rules over one model still cost one call. label_distribution.v1 reads the verdict straight off the winning label, consulting no boundaries at all, and reuses the shared sequence backend with the verdict vocabulary declared inline as the generic classifier signal already does. Rules still matter on the label path even though every rule receives the same verdict, because each carries its own composer and can be gated differently. ConfidenceReported separates a real confidence from the absence of one. The local path and the label contract both report one; score.v1 does not, so the publisher leaves the key absent and the decision engine falls back to its structural default while marking the pool unscored - the treatment keyword, language and pii already receive. Writing a zero instead would silently rank the decision last among scored competitors. The request context reaches the signal, so an abandoned request cancels the outbound call. That meant threading it through the request-fact dispatcher, which had none while the primary dispatcher already carried one. Signed-off-by: theohsiung <theobear870924@gmail.com>
pkg/config/AGENTS.md requires a config-contract change to update the public docs in the same change, so the tutorial, installation configuration, config/README.md and the v0.3 proposal all describe the new surface rather than leaving the schema ahead of its documentation. Two consequences are correct but easy to miss, so the Router says them at startup instead of leaving them to be discovered: score.v1 reports no confidence, so a decision gated on one of those rules ranks on the engine's structural default rather than a reported score - while a local-backed rule beside it still reports one, making the change invisible in the config but visible in which decision wins. label_distribution.v1 has no such gap, and the advisory says so. The hard/easy candidate lists are how the local path produces a score. With a backend they are never read, so editing them has no effect at all; a config mid-migration will legitimately still carry them, which is why this is an advisory rather than an error. The advisories are returned as strings and logged by the validator rather than logged directly, so the reasoning is testable without capturing log output - this package has no precedent for asserting on log lines. Signed-off-by: theohsiung <theobear870924@gmail.com>
**The local path lost its confidence (a regression).** ConfidenceReported was introduced for score.v1 but only ever set on the remote label path, so every config with no backend at all stopped publishing SignalConfidences for complexity: confidence-based decision ranking, numeric predicates on complexity conditions, projection inputs and router-learning outcome scores all silently changed. The field's own doc comment asserted the opposite of what the code did. **The local path ignored the new boundary pair.** It still called classifyComplexityDifficulty(rule.Threshold, ...) and never resolved EffectiveBoundaries, so a rule declaring hard_above/easy_below ran with threshold zero - every positive margin hard, medium unreachable, the declared cut points discarded. The reference config shipped exactly that rule, with a comment claiming it worked. Both paths now share one comparison, resolved once at construction so a malformed pair fails at startup rather than per request. **The validator never ran at config load.** Category registers its contract validator in globalConfigContractValidators; complexity had no equivalent entry, so validate-config tooling, the apiserver and the DSL paths accepted an unresolvable backend and the error only surfaced at classifier construction. The comment claiming otherwise is now true. **threshold was accepted under score.v1.** A remote score arrives in the model's own units, so a symmetric threshold cannot convert it: against a [0,1] scorer it puts everything past the hard cut and makes easy unreachable, while a rule declaring nothing collapses both cut points onto zero. score.v1 now requires the explicit pair. **The DSL dropped the boundary pair.** The decompiler never emitted the four fields and the compiler never read them, so YAML -> DSL -> YAML reverted a rule to threshold semantics. Reading them back through getFloat64Field rather than float32 keeps a declared 0.85 from becoming 0.8500000238418579 - caught by the round-trip test itself. Signed-off-by: theohsiung <theobear870924@gmail.com>
AGENTS.md requires E2E coverage for a behaviour-visible config change, and the gate assigns this one the envoy-ai-gateway profile, which never touches the remote scorer. The profile declares no hard/easy candidates at all. With a backend the local prototype path is unreachable anyway, so a verdict reaching the router can only have come from the remote call - that is what makes the assertion attributable rather than merely consistent. The test then pins two scores and requires them to route differently: 0.90 clears needs_reasoning's boundary but sits inside extreme's medium band, while 0.99 clears both. One remote call, two rules, two verdicts is a shape prototype scoring cannot produce here. The mock scorer is an inline Python server in a ConfigMap on python:3.12-alpine, following mock-category-classifier, so the profile builds no image of its own. It returns exactly one entry because score.v1 rejects anything else, and an unmarked request returns a constant so an assertion cannot go flaky for reasons unrelated to the router. Two things this cost, both worth recording: The router OOMKilled at 2Gi, copied from the category profile. Taking the scorer off-box does not avoid loading the local mmbert embedding model the response cache is configured with; the embedding models are gated separately from the classifier backend. 6Gi, as the jailbreak-onerror profile already found. The model is general-expert rather than a profile-specific name. The shared AIGatewayRoute matches x-ai-eg-model exactly against six names, so anything else has no route and every request 404s at the gateway while the router itself stays healthy and logs a correctly constructed backend - which is exactly how it looks like a router bug and is not one. Also releases the score backend's connector on Classifier.Close. A classifier is rebuilt per recipe and again on every dynamic-config reload, so one that is never closed leaks an idle HTTP transport each time. Signed-off-by: theohsiung <theobear870924@gmail.com>
e155bdd to
a33ac62
Compare
|
Thanks for working on it! |
Xunzhuo
left a comment
There was a problem hiding this comment.
Thanks for the thorough contract work. Four integration blockers remain: remote complexity still constructs and preloads the unused local classifier before attaching the backend, so remote-only startup depends on local model resources and readiness ignores the backend; score.v1 treats a missing or null score as numeric zero; explicit boundaries accept NaN and Inf; and dynamic reload replaces RecipeClassifiers without lifetime-safe retirement, leaking the old remote connectors while only the startup set is registered for shutdown. Please bypass local construction in remote mode with backend-aware readiness, require score presence while preserving valid zero, reject non-finite boundaries, and make reload ownership safe for in-flight snapshots. The exact-head quality job also fails MD012 in the complexity tutorial.
md-fmt enforces MD012/no-multiple-blanks; the section I inserted left two consecutive blank lines before the next heading. Signed-off-by: theohsiung <theobear870924@gmail.com>
vllm-project#3489 renamed providers.defaults.default_model to model and added provider to backend_refs. Merging main brought no textual conflict here, because the profile is a new file - so the stale spelling would have survived and providers.defaults.model would simply have parsed as empty. Verified by loading the profile's config subtree through config.Load rather than by reading the diff. Signed-off-by: theohsiung <theobear870924@gmail.com>
Summary
Complexity could not name a remote model at all —
CategoryModelwas the onlystruct with a
Backendfield. It now reads both remote shapes, because adifficulty model can be built either way and the contract decides how the
runtime reads the answer:
score.v1(new) — a regression model returns one number; each ruleconverts it through its own boundaries. One request, one call, however many
rules read it. Implements the
ScoringBackendtier, declared since thethree-tier split with a note saying nothing wired it in.
label_distribution.v1(reused) — a three-class model returnshard/easy/medium, so the winning label is the verdict and itsprobability is a real confidence.
Both go through the shared connector, so timeout, retry, byte caps and error
mapping stay shared.
Decisions worth knowing
routing.signalsis replacedwholesale per recipe, so a rule-level backend would vanish under any recipe
that did not repeat it.
hard_above+easy_below, orhard_below+easy_above), so there is nodirection/min/maxfield andan overlapping band cannot be written.
thresholdstays the symmetricshorthand.
contractis required for complexity: with two readable shapes, guessingwrong would surface per request rather than at config load. Category, reading
one, keeps the default.
score.v1reports no confidence — a score just short of the boundary isthe least certain position, not a strong one. Validation warns, since it
changes decision ranking.
Full reasoning is in #2921's body.
Validation
make test-semantic-router,recipe-conformance-static, website build, and anew E2E profile run green on kind. The profile declares no local candidates, so
a verdict can only come from the remote call, and one score reaches two
different verdicts through two rules' boundaries.
make dashboard-checkfails on threeevaluationplanetests — verifiedpre-existing at
origin/main; a barego testsilently skips them and reportsok.The E2E model is
general-expertbecause the sharedAIGatewayRoutematchesx-ai-eg-modelagainst six exact names. That is also whycategory-remote-backendhas failed since #3138: filed as #3543.Closes #2921